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Abstract 

Context-dependent rewrite rules are used 
in many areas of natural language and 
speech processing. Work in computa- 
tional phonology has demonstrated that, 
given certain conditions, such rewrite 
rules can be represented as finite-state 
transducers (FSTs). We describe a new 
algorithm for compiling rewrite rules into 
FSTs. We show the algorithm to be sim- 
pler and more efficient than existing al- 
gorithms. Further, many of our appli- 
cations demand the ability to compile 
weighted rules into weighted FSTs, trans- 
ducers generalized by providing transi- 
tions with weights. We have extended 
the algorithm to allow for this. 

1. Motivation 

Rewrite rules are used in many areas of natural 
language and speech processing, including syntax, 
morphology, and phonology 1 . In interesting ap- 
plications, the number of rules can be very large. 
It is then crucial to give a representation of these 
rules that leads to efficient programs. 

Finite-state transducers provide just such a 
compact representation (Mohri, 1994). They are 
used in various areas of natural language and 
speech processing because their increased compu- 
tational power enables one to build very large ma- 
chines to model interestingly complex linguistic 
phenomena. They also allow algebraic operations 
such as union, composition, and projection which 
are very useful in practice (Berstel, 1979; Eilen- 
berg, 1974 1976). And, as originally shown by 
Johnson (1972), rewrite rules can be modeled as 

1 Parallel rewrite rules also have interesting applica- 
tions in biology. In addition to their formal language 
theory interest, systems such as those of Aristid Lin- 
denmayer provide rich mathematical models for bio- 
logical development (Rozenberg and Salomaa, 1980). 
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finite-state transducers, under the condition that 
no rule be allowed to apply any more than a finite 
number of times to its own output. 

Kaplan and Kay (1994), or equivalently Kart- 
tunen (1995), provide an algorithm for compiling 
rewrite rules into finite-state transducers, under 
the condition that they do not rewrite their non- 
contextual part 2 . We here present a new algorithm 
for compiling such rewrite rules which is both sim- 
pler to understand and implement, and computa- 
tionally more efficient. Clarity is important since, 
as pointed out by Kaplan and Kay (1994), the rep- 
resentation of rewrite rules by finite-state trans- 
ducers involves many subtleties. Time and space 
efficiency of the compilation are also crucial. Us- 
ing naive algorithms can be very time consuming 
and lead to very large machines (Liberman, 1994). 

In some applications such as those related 
to speech processing, one needs to use weighted 
rewrite rules, namely rewrite rules to which 
weights are associated. These weights are then 
used at the final stage of applications to output the 
most probable analysis. Weighted rewrite rules 
can be compiled into weighted finite-state trans- 
ducers, namely transducers generalized by pro- 
viding transitions with a weighted output, under 
the same context condition. These transducers 
are very useful in speech processing (Pereira et 
al., 1994). We briefly describe how we have aug- 
mented our algorithm to handle the compilation 
of weighted rules into weighted finite-state trans- 
ducers. 

In order to set the stage for our own contribu- 
tion, we start by reviewing salient aspects of the 
Kaplan and Kay algorithm. 



2 The general question of the decidability of the 
halting problem even for one-rule semi-Thue systems 
is still open. Robert McNaughton (1994) has recently 
made a positive conjecture about the class of the rules 
without self overlap. 
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Figure 1: Compilation of obligatory left-to-right rules, using the KK algorithm. 



2. The KK Algorithm 

The rewrite rules we consider here have the fol- 
lowing general form: 

<t> - ^/A p (2) 

Such rules can be interpreted in the following way: 
<f> is to be replaced by ip whenever it is preceded 
by A and followed by p. Thus, A and p represent 
the left and right contexts of application of the 
rules. In general, <f>, ip, A and p are all regular 
expressions over the alphabet of the rules. Several 
types of rules can be considered depending on their 
being obligatory or optional, and on their direction 
of application, from left to right, right to left or 
simultaneous application. 

Consider an obligatory rewrite rule of the form 

<f> —> tp/X p, which we will assume applies left to 

right across the input string. Compilation of this 
rule in the algorithm of Kaplan and Kay (1994) 
(KK for short) involves composing together six 
transducers, see Figure 1. 

We use the notations of KK. In particular, £ 
denotes the alphabet, < denotes the set of context 
labeled brackets {< a , < c }, > the set {> a , 
> c }, and an additional character representing 
deleted material. Subscript symbols of an expres- 
sion are symbols which are allowed to freely ap- 
pear anywhere in the strings represented by that 
expression. Given a regular expression r, Id(r) is 
the identity transducer obtained from an automa- 
ton A representing r by adding output labels to A 
identical to its input labels. 

The first transducer, Prologue, freely intro- 
duces labeled brackets from the set {< a , 
<c; >a, >i, >c} which are used by left and 
right context transducers. The last transducer, 
Prologue -1 , erases all such brackets. 

In such a short space, we can of course not 
hope to do justice to the KK algorithm, and the 
reader who is not familiar with it is urged to con- 
sult their paper. However, one point that we do 
need to stress is the following: while the con- 
struction of Prologue, Prologue -1 and Replace 



is fairly direct, construction of the other transduc- 
ers is more complex, with each being derived via 
the application of several levels of regular oper- 
ations from the original expressions in the rules. 
This clearly appears from the explicit expressions 
we have indicated for the transducers. The con- 
struction of the three other transducers involves 
many operations including: two intersections of 
automata, two distinct subtractions, and nine 
complementations. Each subtraction involves an 
intersection and a complementation algorithm 3 . 
So, in the whole, four intersections and eleven 
complementations need to be performed. 

Intersection and complementation are classi- 
cal automata algorithms (Aho et al., 1974; Aho 
et al., 1986). The complexity of intersection is 
quadratic. But the classical complementation al- 
gorithm requires the input automaton to be de- 
terministic. Thus, each of these 11 operations re- 
quires first the determinization of the input. Such 
operations can be very costly in the case of the 
automata involved in the KK algorithm 4 . 

In the following section we briefly describe a 
new algorithm for compiling rewrite rules. For rea- 
sons of space, we concentrate here on the com- 
pilation of left-to-right obligatory rewrite rules. 
However, our methods extend straightforwardly to 
other modes of application (optional, right-to-left, 
simultaneous, batch), or kinds of rules (two-level 
rules) discussed by Kaplan and Kay (1994). 

3 A subtraction can of course also be performed di- 
rectly by combining the two steps of intersection and 
complementation, but the corresponding algorithm 
has exactly the same cost as the total cost of the two 
operations performed consecutively. 

4 One could hope to find a more efficient way of de- 
termining the complement of an automaton that would 
not require determinization. However, this problem 
is PSPACE-complete. Indeed, the regular expression 
non-universality problem is a subproblem of comple- 
mentation known to be PSPACE-complete (Garey and 
Johnson, 1979, page 174), (Stockmeyer and Meyer, 
1973). This problem also known as the emptiness 
of complement problem has been extensively studied 
(Aho et al., 1974, page 410-419). 



3. New Algorithm 

3.1. Overview 

In contrast to the KK algorithm which introduces 
brackets everywhere only to restrict their occur- 
rence subsequently, our algorithm introduces con- 
text symbols just when and where they are needed. 
Furthermore, the number of intermediate trans- 
ducers necessary in the construction of the rules 
is smaller than in the KK algorithm, and each of 
the transducers can be constructed more directly 
and efficiently from the primitive expressions of 
the rule, <f>, t/>, A, p. 

A transducer corresponding to the left-to- 
right obligatory rule <f> — ► ip/A p can be ob- 
tained by composition of five transducers: 

r o / o replace 0^0/2 (3) 

1. The transducer r introduces in a string a 
marker > before every instance of p. For rea- 
sons that will become clear we will notate this 
as SV S* > p. 

2. The transducer / introduces markers <i and 
<2 before each instance of <f> that is followed 
by >: (SU {>})*</> >^ (EU {>})*{<!, <2 
}(/)>. In other words, this transducer marks 
just those <f> that occur before p. 

3. The replacement transducer replace replaces 
<f> with ip in the context <i <f> >, simultane- 
ously deleting > in all positions (Figure 2). 
Since >, <i, and <2 need to be ignored when 
determining an occurrence of <f>, there are 
loops over the transitions >: e,<\. e,<2'. e 
at all states of <f>, or equivalently of the states 
of the cross product transducer <f> x t/>. 

4. The transducer l\ admits only those strings 
in which occurrences of <i are preceded 
by A and deletes <i at such occurrences: 
S*A<i^S*A. 

5. The transducer I2 admits only those strings 
in which occurrences of <2 are not preceded 
by_A and deletes <2 at such occurrences: 
S*A< 2 ^S*A. 

Clearly the composition of these transducers leads 
to the desired result. The construction of the 
transducer replace is straightforward. In the fol- 
lowing, we show that the construction of the other 
four transducers is also very simple, and that it 
only requires the determinization of 3 automata 
and additional work linear (time and space) in the 
size of the determinized automata. 

3.2. Markers 
Markers of TYPE 1 

Let us start by considering the problem of con- 
structing what we shall call a TYPE 1 transducer, 
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Figure 2: Replacement transducer replace in the 
obligatory left-to-right case. 

which inserts a marker after all prefixes of a string 
that match a particular regular expression. Given 
a regular expression [3 defined on the alphabet S, 
one can construct, using classical algorithms (Aho 
et al., 1986), a deterministic automaton a repre- 
senting £*/?. As with the KK algorithm, one can 
obtain from a a transducer x = Id (a) simply by 
assigning to each transition the same output label 
as the input label. We can easily transform x into 
a new transducer r such that it inserts an arbi- 
trary marker # after each occurrence of a pattern 
described by [3. To do so, we make final the non- 
final states of x an d for any final state 5 of x we 
create a new state q' , a copy of q. Thus, q' has 
the same transitions as q, and q' is a final state. 
We then make q non-final, remove the transitions 
leaving q and add a transition from q to q' with 
input label the empty word e, and output Fig- 
ures 3 and 4 illustrate the transformation of x into 
r. 




Figure 3: Final state q of x with entering and 
leaving transitions. 




Figure 4: States and transitions of r obtained by 
modifications of those of x- 



Proposition 1 Let a be a deter mint site automa- 
ton representing T,*/3, then the transducer r ob- 
tained as described above is a transducer post- 
marking occurrences of [3 in a string of S* by 

Proof. The proof is based on the observa- 
tion that a deterministic automaton representing 
is necessarily complete 5 . Notice that non- 
deterministic automata representing are not 
necessarily complete. Let 5 be a state of a and let 
u G S* be a string reaching q 6 . Let v be a string 
described by the regular expression [3. Then, for 
any a G £, uav is in £*/?. Hence, i/ai> is accepted 
by the automaton a, and, since a is deterministic, 
there exists a transition labeled with a leaving q. 
Thus, one can read any string u G S* using the 
automaton a. Since by definition of a, the state 
reached when reading a prefix u' of u is final iff 
u' G £*/?, by construction, the transducer r in- 
serts the symbol # after the prefix «' iff «' ends 
with a pattern of [3. This ends the proof of the 
proposition. □ 

Markers of TYPE 2 

In some cases, one wishes to check that any 
occurrence of # in a string s is preceded (or fol- 
lowed) by an occurrence of a pattern of [3. We 
shall say that the corresponding transducers are 
of TYPE 2. They play the role of a filter. Here 
again, they can be defined from a deterministic au- 
tomaton representing £*/?. Figure 5 illustrates the 
modifications to make from the automaton of fig- 
ure 3. The symbols # should only appear at final 
states and must be erased. The loop # : e added 
at final states of Ld(a) is enough for that purpose. 
All states of the transducer are then made final 
since any string conforming to this restriction is 
acceptable: cf. the transducer l\ for A above. 




Figure 5: Filter transducer, TYPE 2. 



5 An automaton A is complete iff at any state q and 
for any element a of the alphabet E there exists at least 
one transition leaving q labeled with a. In the case of 
deterministic automata, the transition is unique. 

6 We assume all states of a accessible. This is true 
if a is obtained by determinization. 



Markers of TYPE 3 

In other cases, one wishes to check the reverse 
constraint, that is that occurrences of # in the 
string s are not preceded (or followed) by any oc- 
currence of a pattern of [3. The transformation 
then simply consists of adding a loop at each non- 
final state of Ld(a), and of making all states final. 
Thus, a state such as that of figure 6 is trans- 




Figure 6: Non-final state q of a. 

formed into that of figure 5. We shall say that the 
corresponding transducer is of TYPE 3: cf. the 
transducer I2 for A. 

The construction of these transducers (type 
1-3) can be generalized in various ways. In par- 
ticular: 

• One can add several alternative markers 
{#1, • • • , after each occurrence of a pat- 
tern of [3 in a string. The result is then an 
automaton with transitions labeled with, for 
instance, #1, • • • , after each pattern of [3: 
cf. transducer / for <f> above. 

• Instead of inserting a symbol, one can delete 
a symbol which would be necessarily present 
after each occurrence of a pattern of [3. 

For any regular expression a, de- 
fine Marker(a, type, deletions, insertions) as the 
transducer of type type constructed as previously 
described from a deterministic automaton repre- 
senting a, insertions and deletions being, respec- 
tively, the set of insertions and deletions the trans- 
ducer makes. 

Proposition 2 For any regular expression 
a, Marker(a, type, deletions, insertions) can be 
constructed from a deterministic automaton rep- 
resenting a in linear time and space with respect 
to the size of this automaton. 

Proof. We proved in the previous proposition that 
the modifications do indeed lead to the desired 
transducer for TYPE 1. The proof for other cases 
is similar. That the construction is linear in space 
is clear since at most one additional transition and 
state is created for final or non-final states 7 . The 
overall time complexity of the construction is lin- 
ear, since the construction of Id(a) is linear in the 

r For TYPE 2 and TYPE 3, no state is added but only 
a transition per final or non-final state. 



r = [reverse(Marker(T,*reverse(p), 1, {>}, 0))] (4) 

/ = [reverse(Marker((T, U {>})* reverse((f) > >), 1, {<i, <2}, 0))] (5) 

h = [Marker^* X, 2,0, {<i})]< 2:<2 (6) 

h = [Marker^* X,3,®,{< 2 })} (7) 

Figure 7: Expressions of the r, /, l\, and / 2 using Marker. 



number of transitions of a and that other modifi- 
cations consisting of adding new states and transi- 
tions and making states final or not are also linear. 
□ 

We just showed that Marker (a, type, de- 
letions, insertions) can be constructed in a very 
efficient way. Figure 7 gives the expressions of the 
four transducers r, /, l\, and / 2 using Marker. 

Thus, these transducers can be constructed 
very efficiently from deterministic automata repre- 
senting 8 Y<*reverse(p), (S U {>})* reversely >), 
and S*A. The construction of r and / requires 
two reverse operations. This is because these two 
transducers insert material before p or <f>. 

4. Extension to Weighted Rules 

In many applications, in particular in areas re- 
lated to speech, one wishes not only to give all 
possible analyses of some input, but also to give 
some measure of how likely each of the analyses is. 
One can then generalize replacements by consid- 
ering extended regular expressions, namely, using 
the terminology of formal language theory, ratio- 
nal power series (Berstel and Reutenauer, 1988; 
Salomaa and Soittola, 1978). 

The rational power series we consider here are 
functions mapping S* to 1Z+ U {oo} which can be 
described by regular expressions over the alphabet 
(TZ+ U {oo}) x S. S = (4a) (2b)* (3b) is an example 
of rational power series. It defines a function in 
the following way: it associates a non-null num- 
ber only with the strings recognized by the regu- 
lar expression ab*b. This number is obtained by 
adding the coefficients involved in the recognition 
of the string. The value associated with abbb, for 
instance, is (S, abbb) = 4 + 2 + 2 + 3 = 11. 

In general, such extended regular expressions 
can be redundant. Some strings can be matched 



8 As in the KK algorithm we denote by c/>> the set 
of the strings described by <f> containing possibly oc- 
currences of > at any position. In the same way, sub- 
scripts such as >:> for a transducer r indicate that 
loops by >:> are added at all states of r. We de- 
note by reverse(a) the regular expression describing 
exactly the reverse strings of a if a is a regular expres- 
sion, or the reverse transducer of a if a is a transducer. 



in different ways with distinct coefficients. The 
value associated with those strings is then the min- 
imum of all possible results. 5" = (2a)(36)(46) + 
(5a) (36*) matches abb with the different weights 
2 + 3 + 4 = 9 and 5 + 3 + 3 = 11. The mini- 
mum of the two is the value associated with abb: 
(S',abb) = 9. Non-negative numbers in the defi- 
nition of these power series are often interpreted 
as the negative logarithm of probabilities. This 
explains our choice of the operations: addition of 
the weights along the string recognition and min, 
since we are only interested in that result which 
has the highest probability 9 . 

Rewrite rules can be generalized by letting ip 
be a rational power series. The result of the ap- 
plication of a generalized rule to a string is then 
a set of weighted strings which can be represented 
by a weighted automaton. Consider for instance 
the following rule, which states that an abstract 
nasal, denoted N , is rewritten as m in the context 
of a following labial: 

N mj [+labial] (8) 

Now suppose that this is only probabilistically 
true, and that while ninety percent of the time 
N does indeed become m in this environment, 
about ten percent of the time in real speech it be- 
comes n. Converting from probabilities to weights, 
one would say that N becomes m with weight 
a = — log(0.9), and n with weight [3 = — log(O.l), 
in the stated environment. One could represent 
this by the following rule: 

N —> am + j3n/ [+labial] (9) 

We define Weighted finite-state transducers as 
transducers such that in addition to input and out- 
put labels, each transition is labeled with a weight. 

The result of the application of a weighted 
transducer to a string, or more generally to an 
automaton is a weighted automaton. The corre- 
sponding operation is similar to the unweighted 
case. However, the weight of the transducer 
and those of the string or automaton need to 
be combined too, here added, during composition 
(Pereira et al., 1994). 

9 Using the terminology of the theory of languages, 
the functions we consider here are power series de- 
fined on the tropical semiring (TZ+ U{oo}, min, +, oo, 0) 
(Kuich and Salomaa, 1986). 




Figure 8: Transducer representing the rule 9. 

We have generalized the composition opera- 
tion to the weighted case by introducing this com- 
bination of weights. The algorithm we described 
in the previous sections can then also be used to 
compile weighted rewrite rules. 

As an example, the obligatory rule 9 can be 
represented by the weighted transducer of Fig- 
ure 8 10 . The following theorem extends to the 
weighted case the assertion proved by Kaplan and 
Kay (1994). 

Theorem 1 A weighted rewrite rule of the type 
defined above that does not rewrite its non- 
contextual part can be represented by a weighted 
finite-state transducer. 

Proof. The construction we described in the pre- 
vious section also provides a constructive proof 
of this theorem in the unweighted case. In case 
ip is a power series, one simply needs to use in 
that construction a weighted finite-state trans- 
ducer representing t/>. By definition of composition 
of weighted transducers, or multiplication of power 
series, the weights are then used in a way consis- 
tent with the definition of the weighted context- 
dependent rules. □ 

5. Experiments 

In order to compare the performance of the al- 
gorithm presented here with KK, we timed both 
algorithms on the compilation of individual rules 
taken from the following set (k £ [0, 10]): 

a^b/c k (10) 

a->b/ c k (11) 

10 We here use the symbol @ to denote all letters 
different from b, m, n, p, and N. 



In other words we tested twenty two rules where 
the left context or the right context is varied in 
length from zero to ten occurrences of c. For our 
experiments, we used the alphabet of a realistic 
application, the text analyzer for the Bell Labora- 
tories German text-to-speech system consisting of 
194 labels. All tests were run on a Silicon Graph- 
ics IRIS Indigo 4000, 100 MhZ IP20 Processor, 
128 Mbytes RAM, running IRIX 5.2. Figure 9 
shows the relative performance of the two algo- 
rithms for the left context: apparently the per- 
formance of both algorithms is roughly linear in 
the length of the left context, but KK has a worse 
constant, due to the larger number of operations 
involved. Figure 10 shows the equivalent data for 
the right context. At first glance the data looks 
similar to that for the left context, until one no- 
tices that in Figure 10 we have plotted the time on 
a log scale: the KK algorithm is hyperexponential. 

What is the reason for this performance degra- 
dation in the right context? The culprits turn 
out to be the two intersectands in the expression 
of Rightcontext(p, <,>) in Figure 1. Consider 
for example the righthand intersectand, namely 

^>o > / 9 >o^>o — > ^>oi which is the complement 
of S^g > /9>o£>o — > ^>o- A s previously in- 
dicated, the complementation algorithm requires 
determinization, and the determinization of au- 
tomata representing expressions of the form S*a, 
where a is a regular expression, is often very ex- 
pensive, specially when the expression a is already 
complex, as in this case. 

Figure 11 plots the behav ior of determiniza- 
tion on the expression S^g > /9 > oSJ >0 — > S^g 

for each of the rules in the set a — ► bj c k , 

(k £ [0, 10]). On the horizontal axis is the num- 
ber of arcs of the non-deterministic input machine, 
and on the vertical axis the log of the number of 
arcs of the deterministic machine, i.e. the ma- 
chine result of the determinization algorithm with- 
out using any minimization. The perfect linearity 
indicates an exponential time and space behav- 
ior, and this in turn explains the observed differ- 
ence in performance. In contrast, the construction 
of the right context machine in our algorithm in- 
volves only the single determinization of the au- 
tomaton representing S* p, and thus is much less 
expensive. The comparison just discussed involves 
a rather artificial ruleset, but the differences in 
performance that we have highlighted show up in 
real applications. Consider two sets of pronun- 
ciation rules from the Bell Laboratories German 
text-to-speech system: the size of the alphabet for 
this ruleset is 194, as noted above. The first rule- 
set, consisting of pronunciation rules for the ortho- 
graphic vowel <6> contains twelve rules, and the 
second ruleset, which deals with the orthographic 
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Figure 9: Compilation times for rules of the form 
a hj c k , (k £ [0, 10]). 

vowel <a> contains twenty five rules. In the ac- 
tual application of the rule compiler to these rules, 
one compiles the individual rules in each ruleset 
one by one, and composes them together in the 
order written, compacts them after each composi- 
tion, and derives a single transducer for each set. 
When done off-line, these operations of compo- 



Table 1: Comparison in a real example. 



Rules 


KK 


New 




time 

(s) 


space 


time 
(s) 


space 


states 


arcs 


states 


arcs 


<6> 


62 


412 


50,475 


47 


394 


47,491 


<a> 


284 


1,939 


215,721 


240 


1,927 


213,408 



sition and compaction dominate the time corre- 
sponding to the construction of the transducer for 
each individual rule. The difference between the 
two algorithms appears still clearly for these two 
sets of rules. Table 1 shows for each algorithm 
the times in seconds for the overall construction, 
and the number of states and arcs of the output 
transducers. 

6. Conclusion 

We briefly described a new algorithm for compiling 
context-dependent rewrite rules into finite-state 
transducers. Several additional methods can be 
used to make this algorithm even more efficient. 

The automata determinizations needed for 
this algorithm are of a specific type. They repre- 



8 - 




2 4 6 8 10 

Length of Right Context 

Figure 10: Compilation times for rules of the form 
a hj c k , (k £ [0, 10]). 



sent expressions of the type T,*(f> where <f> is a reg- 
ular expression. Given a deterministic automaton 
representing <f>, such determinizations can be per- 
formed in a more efficient way using failure func- 
tions (Mohri, 1995). Moreover, the corresponding 
determinization is independent of £ which can be 
very large in some applications. It only depends 
on the alphabet of the automaton representing <f>. 

One can devise an on-the-fly implementation 
of the composition algorithm leading to the final 
transducer representing a rule. Only the neces- 
sary part of the intermediate transducers is then 
expanded for a given input (Pereira et al., 1994). 

The resulting transducer representing a rule 
is often subsequentiable or p-subsequentiable. It 
can then be determinized and minimized (Mohri, 
1994). This both makes the use of the transducer 
time efficient and reduces its size. 

We also indicated an extension of the theory 
of rule-compilation to the case of weighted rules, 
which compile into weighted finite-state transduc- 
ers. Many algorithms used in the finite-state the- 
ory and in their applications to natural language 
processing can be extended in the same way. 

To date the main serious application of this 
compiler has been to developing text-analyzers 
for text-to-speech systems at Bell Laboratories 
(Sproat, 1996): partial to more-or-less complete 
analyzers have been built for Spanish, Italian, 
French, Romanian, German, Russian, Mandarin 
and Japanese. However, we hope to also be able to 
use the compiler in serious applications in speech 
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Figure 11: Number of arcs in the non- 
deterministic automaton r representing PS = 
^>o > / 9 >o^>o — > ^>o versus the log of the num- 
ber of arcs in the automaton obtained by deter- 
minization of r. 

recognition in the future. 
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